{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cf641fa2",
   "metadata": {},
   "source": [
    "# 2.2 数据预处理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "842833d0",
   "metadata": {},
   "source": [
    "   问题1：为什么要引入数据预处理\n",
    "    \n",
    "    · 个人对数据预处理的理解，在普通的机器学习中，比如红酒模型的应用前，需要对数据进行一个简单的处理，把一些非线性的数据转化为可处理的线性数据，进行分类存储，以便后续训练和测试时可以便捷的使用"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e366fcd",
   "metadata": {},
   "source": [
    "2.2.1 读取数据库\n",
    "\n",
    "    · 接下来以创建一个人工数据集为例，并存储到csv文件../data/house_tiny.csv中，其它数据格式的存储可以参考#\n",
    "    · mkdir_if_not_exist函数可确保目录../data存在\n",
    "    · 标记下方的函数、类或语句将保存在 d2l 软件包中，以便以后可以直接调用它们（例如 d2l.mkdir_if_not_exist(path)）无需重新定义"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e3ab35d",
   "metadata": {},
   "source": [
    "例1：建立文件，写入数据，打印数据，修改数据，转换为张量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2d4db16c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.makedirs(os.path.join('..', 'data'),exist_ok=True)\n",
    "data_file = os.path.join('..', 'data', 'house_tiny.csv')\n",
    "with open(data_file, 'w') as f:\n",
    "    f.write('NumRooms, Alley, Price\\n')  # 列名\n",
    "    f.write('NA, Pave, 127500\\n')  # 每行表示一个数据样本（sample）\n",
    "    f.write('2, NA, 106000\\n')\n",
    "    f.write('4, NA, 178000\\n')\n",
    "    f.write('NA, NA, 140000\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9293e3b3",
   "metadata": {},
   "source": [
    "    · 接下来从创建好的csv文件中加载原始数据集，\n",
    "    · 导入pandas（提前在环境中安装或 ！pip install pandas）及调用read_csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e5e0168a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   NumRooms  Alley   Price\n",
      "0       NaN   Pave  127500\n",
      "1       2.0     NA  106000\n",
      "2       4.0     NA  178000\n",
      "3       NaN     NA  140000\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv(data_file)\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b0d8b0f",
   "metadata": {},
   "source": [
    "    · 上述 Numrooms表示房间数量，Alley表示巷子类型，Price表示房子售价\n",
    "    · cd ~/opt/anaconda3/envs/pytorch/lib/python3.7/site-packages/pandas/io/parsers.py 如果在写入文件时，忘记了，再次写入可能会报错，\n",
    "    · 利用 rm parsers.py 删除这个文件，再次运行即可"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b89503cd",
   "metadata": {},
   "source": [
    "2.2.2 处理缺失值\n",
    "\n",
    "    · 代码中，’NaN‘项表示缺失值\n",
    "    · 为了处理缺失值，日常采用 插值 或 删除，本次采用插值方案 \n",
    "    · 通过位置索引iloc，我们将 data 分成 inputs 和 outputs，其中前者为 data的前两列，而后者为 data的最后一列。对于 inputs 中缺少的数值，我们用同一列的均值替换 “NaN” 项"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "687f661e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   NumRooms  Alley\n",
      "0       3.0   Pave\n",
      "1       2.0     NA\n",
      "2       4.0     NA\n",
      "3       3.0     NA\n"
     ]
    }
   ],
   "source": [
    "inputs, outputs = data.iloc[:, 0:2], data.iloc[:, 2]\n",
    "inputs = inputs.fillna(inputs.mean()) # 函数mean()计算均值\n",
    "print(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cf3caf6",
   "metadata": {},
   "source": [
    "    · 对于 inputs 中的类别值或离散值，我们将 “NaN” 视为一个类别。\n",
    "    · 由于 “巷子”（“Alley”）列只接受两种类型的类别值 “Pave” 和 “NaN”，pandas 可以自动将此列转换为两列       \n",
    "    “Alley_Pave” 和 “Alley_nan”。\n",
    "    · 子类型为 “Pave” 的行会将“Alley_Pave”的值设置为1，“Alley_nan”的值设置为0。\n",
    "    · 少巷子类型的行会将“Alley_Pave”和“Alley_nan”分别设置为0和1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c840f488",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   NumRooms   Alley_ NA   Alley_ Pave\n",
      "0       3.0           0             1\n",
      "1       2.0           1             0\n",
      "2       4.0           1             0\n",
      "3       3.0           1             0\n"
     ]
    }
   ],
   "source": [
    "inputs = pd.get_dummies(inputs, dummy_na=False)\n",
    "print(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7e9a54c",
   "metadata": {},
   "source": [
    "    · 上述pd.get_dummies函数的参数中，\n",
    "    · 官方文档：\n",
    "    · dummy_na  bool, default False\n",
    "    · Add a column to indicate NaNs, if False NaNs are ignored."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5afe89d",
   "metadata": {},
   "source": [
    "2.2.3 转换为张量格式\n",
    "\n",
    "    · 目前数据inputs和outputs中所有的条目都是数值类型\n",
    "    · 将其转换为张量格式后，可以对其进行张量函数的操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8eb1c6e2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[3., 0., 1.],\n",
       "         [2., 1., 0.],\n",
       "         [4., 1., 0.],\n",
       "         [3., 1., 0.]], dtype=torch.float64),\n",
       " tensor([127500, 106000, 178000, 140000]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch \n",
    "\n",
    "X, y = torch.tensor(inputs.values), torch.tensor(outputs.values)\n",
    "X, y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceeaae8c",
   "metadata": {},
   "source": [
    "2.2.4 小结\n",
    "\n",
    "    · 像panda这样的包，在python中也有许多，其也可以与张量兼容\n",
    "    · 插值和删值均可以处理缺失的数据，如果缺少过多，又无法通过均值来处理，\n",
    "    · 可以试着把对应行或者列删除"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a0f6438",
   "metadata": {},
   "source": [
    "2.2.5 练习\n",
    "    \n",
    "    · 创建包含更多行和列的原始数据集。\n",
    "\n",
    "    1、删除缺失值最多的列。\n",
    "    2、将预处理后的数据集转换为张量格式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3728604e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    " \n",
    "os.makedirs(os.path.join('..', 'data'), exist_ok=True)  \n",
    "# '..' notebooks的下一级目录\n",
    "data_file = os.path.join('..', 'data', 'fruit_category.csv')\n",
    "with open(data_file, 'w') as f:\n",
    "    f.write('Name, color, Price, fav\\n') #列名\n",
    "    f.write('apple, red, 5.0, 4\\n') # a single price\n",
    "    f.write('banana, NA, NA, 5\\n')\n",
    "    f.write('Na, orange, 2.5, 4\\n')\n",
    "    f.write('watermelon, NA, NA, 3\\n')\n",
    "    f.write('NA, white, NA, 4\\n')\n",
    "    f.write('NA, yellow, NA, 2\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6d9c52dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name      2\n",
      " color    0\n",
      " Price    0\n",
      " fav      0\n",
      "dtype: int64\n",
      "0    0\n",
      "1    0\n",
      "2    0\n",
      "3    0\n",
      "4    1\n",
      "5    1\n",
      "dtype: int64\n",
      "         Name    color  Price   fav\n",
      "0       apple      red    5.0     4\n",
      "1      banana       NA     NA     5\n",
      "2          Na   orange    2.5     4\n",
      "3  watermelon       NA     NA     3\n",
      "4         NaN    white     NA     4\n",
      "5         NaN   yellow     NA     2\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv(data_file)\n",
    "# 统计每列数据缺失值的分布情况\n",
    "print(data.isnull().sum(0))\n",
    "\n",
    "#统计每行数据缺失值的分布情况\n",
    "#通过指定参数axis=1来实现对每行数据的缺失值进行统计，\n",
    "# 默认是axis=0表示列\n",
    "print(data.isnull().sum(axis=1))\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "66424af9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         Name    color   fav\n",
      "0       apple      red     4\n",
      "1      banana       NA     5\n",
      "2          Na   orange     4\n",
      "3  watermelon       NA     3\n",
      "4         NaN    white     4\n",
      "5         NaN   yellow     2\n"
     ]
    }
   ],
   "source": [
    "data = data.drop(data.columns[2],axis=1) \n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9a97271c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         Name    color\n",
      "0       apple      red\n",
      "1      banana       NA\n",
      "2          Na   orange\n",
      "3  watermelon       NA\n",
      "4         NaN    white\n",
      "5         NaN   yellow\n"
     ]
    }
   ],
   "source": [
    "inputs, outputs = data.iloc[:, 0:2], data.iloc[:, 2] # 删除最后一列后，没有输出\n",
    "inputs = inputs.fillna(inputs.mean())\n",
    "print(inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1675551b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Name_Na  Name_apple  Name_banana  Name_watermelon   color_ NA  \\\n",
      "0        0           1            0                0           0   \n",
      "1        0           0            1                0           1   \n",
      "2        1           0            0                0           0   \n",
      "3        0           0            0                1           1   \n",
      "4        0           0            0                0           0   \n",
      "5        0           0            0                0           0   \n",
      "\n",
      "    color_ orange   color_ red   color_ white   color_ yellow  \n",
      "0               0            1              0               0  \n",
      "1               0            0              0               0  \n",
      "2               1            0              0               0  \n",
      "3               0            0              0               0  \n",
      "4               0            0              1               0  \n",
      "5               0            0              0               1  \n"
     ]
    }
   ],
   "source": [
    "inputs = pd.get_dummies(inputs, dummy_na=False)\n",
    "print(inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4b2d9777-611a-42c2-8837-18bbbfb36bf8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[0., 1., 0., 0., 0., 0., 1., 0., 0.],\n",
       "         [0., 0., 1., 0., 1., 0., 0., 0., 0.],\n",
       "         [1., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
       "         [0., 0., 0., 1., 1., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 1., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 1.]]),\n",
       " tensor([4., 5., 4., 3., 4., 2.]))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, Y = torch.tensor(inputs.values).to(torch.float32), torch.tensor(outputs.values).to(torch.float32)\n",
    "X, Y"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
